The integration of Internet services and mobile communications has been an important evolutionary step in the history of communications services. As the demand for mobile network services (including voice and packet-based IP data) increases, so does the need to characterize (by throughput, delay, delay variation, service availability, packet loss rates, reliability and priority measurements) the Quality of Service (QoS) provided by 2G, 2.5G, and 3G network services. Such networks utilize a wide variety of heterogeneous protocols (e.g., WAP, GSM, UMTS, hypertext transfer protocol/HTTP, short messaging service/SMS, General Packet Radio Services/GPRS, etc.) to provide connectivity between an end user and the services desired.
FIG. 1 illustrates the physical layout of a system framework that may be used to test the QoS of a service provided on a network 100. Operators at a service operations center (SOC) or network operations center (NOC) 102 in communication with a wireless QoS manager 104 such as, for example, the Agilent Technologies™ Wireless QoS Manager create structured network QoS tests. QoS is often characterized with active testing, wherein one or more test probes 106 simulate user operation of a device attempting to interact with the service (shown as provided by servers 101 through a gateway 103) and take measurements while doing so. The active test probes are controlled by an Active Test Controller (ATC) 108 in communication with the manager 104.
FIG. 2 reflects a graphical representation of a QoS model that describes monitored services and shows how they are organized. As suggested by the hierarchical arrangement, the test code to perform the QoS testing required to characterize network service(s) must often be executed in a structured sequence 110 of test modules 112 because interaction with the network service requires establishing particular “states” such as, for example, initializing a device, establishing a connection, or logging into a server.
Errors that occur while a client device (simulated by a test probe 106) is in a particular state can result in a number of undesirable events. For example, if a client fails to ‘log out’ from a session account before ending its test, the next test may be denied access to that same account. Several different approaches to handling the occurrence of errors during execution of network services testing have been undertaken. Some simply abort a test that experiences an error, leaving the service (or servers) in an unknown state that can introduce errors for subsequent tests utilizing the service. Some methods execute tests utilizing a “test step” approach, but leave error handling to the individual tests steps, which may succeed in certain circumstances, but once a state has been established by a test step, a subsequent test step has no clear and simple mechanism to call back into the prior state-creating step to reset the state. More complex error handling routines have also been written and customized, but require knowledge of the precise test sequence to be executed in order to be able to restore the state from whatever point during the test that the error occurs.
Network management often involves monitoring distinct network sub-systems (e.g., GPRS) by network operators through the use of API's. If a breakdown in the network occurs, many conventional ATCs 108 will fail all the tests scheduled for execution after the test in which the error occurred, as the ATC will follow the prescribed test sequence undeterred by the error. Thus, a multitude of error messages may be sent to monitors of various aspects of the network that might actually be operating correctly. Even potentially transient error messages can have negative effects on network QoS monitoring, confusing the error identification process and possibly resulting in the powering off of properly functioning network elements.
Whatever the benefits of the error handling approaches and testing systems discussed above, recovery from errors encountered during network testing remains disruptive, and since such systems do not leverage existing test operation modules in an automated manner, they can be costly in terms of the resources required to more precisely identify sources of the errors encountered.